A. Field of Invention
The invention relates to the apparatus, method, system, and applications of interfacing computers, various electronic devices, and/or other systems with people using gesture or body, or body parts movements based controls which in some embodiments can be structured as a language, and which are interpreted by the interface which in a particular case can contain a depth-map generator device, which may be a specialized 3D (three-dimensional) image generating camera and a processing device to detect, identify, and understand the gesture command.
B. Description of the Related Art
The field of 3D objects reconstruction has been investigated thoroughly by the academia and industry and results of the efforts put so far in this field are abundant. Hereafter a brief overview is given. Various methods are known in the art for 3D object reconstructions and/or depth measurement of points and surfaces of real objects using image capturing devices, controlled light environment or structured light generators, and computers. Named also optical 3D mapping, the generation of a 3D profile of the surface of an object by processing the optical image of that object has been for a long time the subject of numerous scientific works and patents. There is a vast literature on the subject of depth mapping methods. The most straight forward scheme combines a distance measuring device with a scanning method and system. In this category are 3D laser micro-vision] for micro objects and laser scanners for macro objects.
Another approach that has been studied extensively is depth mapping based on triangulation. Triangulation methods require mathematical manipulation in order to obtain the depth information. Another interesting scheme for depth mapping involves the projection of structured patterns. A structured pattern is projected over an object and from the distortion of the projected pattern, the surface shape of the object is determined. Various kinds of projected patterns have been used, including stripe, sinusoidal, coded, spatially intensity modulated, pseudo-random grid, and Moiré patterns.
All-solid-state imaging range cameras were proposed in technical publications or patents like U.S. Pat. Nos. 4,687,325, or 5,682,229 and others. The Time-of-Flight method was used to produce a depth map of the object by sending invisible optical radiation on the object and capturing with an image capturing device the reflected radiation on which a processing device calculated the time needed by the optical radiation to travel to the object and for the reflected image to be captured by the image capturing device, the optical radiation emitting device and the capturing device being practically mounted together on the same location on the apparatus. The following patents were filled in U.S. Pat. Nos. 7,224,384 B1, 7,671,391 and others. Some technologies were produced and patented for various types of image capturing devices such as the U.S. Pat. No. 7,688,944 B2.
Some methods are based on generating a precisely timed beam of energy emitted from a single source with the beam illuminating the scene, while the reflected energy from the beam is detected with a camera whose sensitivity is precisely timed and synchronized with the emitted beam. The reflected energy is separated and segregated according to the time of arrival at the camera as for example in the U.S. Pat. No. 5,081,530. One or more cycles of energy are emitted; the camera segregates reflected energy during each cycle by separately storing the energy detected before and after a given time. The ratio of two consecutive separately detected energies conveys depth or third dimension information which is processed according to a special algorithm. Signals can be displayed to a viewer to create a stereoscopic image or used by a machine for automatic response to a three dimensional environment.
Some methods, and system were based on projecting a grid of a randomly organized speckle of dots, called a Pseudo-Random Binary Array (PRBA) which is generated by a light source traversing a transparent plastic sheet on which the PRBA was imprinted. The depth of the points on the object surface is calculated by processing the information contained in the dots present on the object image acquired by an image acquisition device and a reference image of a calibrated PRBA pattern imprinted on the transparent plastic sheet. A coarse disparity map is generated by the image processing algorithm which is called a coarse (3D) depth map. Using the epipolar geometry, the camera model and the Fundamental Matrix a stereo fusion algorithm based on Dynamic Programming technique combined with autoregressive modeling allows to reconstruct a 3D epipolarized image of the object with high precision (in a range of tens of tau). A de-epipolarization method is finally providing the real 3D image of the object. The method is repetitively applied until all 3D views of the object are reconstructed. This technique is prior art for the U.S. Pat. No. 6,751,344, Patent Applications US 2008/0240602, U.S. Provisional Patent Application 61/016,832, U.S. patent application Ser. No. 11/899,542, U.S. Provisional Patent Application 60/909,487, and International Publication WO 2007/043036.
Some other methods and system also published were based on colored PRBA grid of randomly organized and differently colored lines which were projected on a real object, the grid being generated by a light source traversing a transparent plastic sheet on which the colored PRBA grid was imprinted. An image acquisition unit detects the light response of the illuminated region of the object and generates image data. Deformations of the colored grid pattern as projected on the object are compared with a reference image of the pattern as projected on the real object, in real-time and from the above comparison a set of control points is obtained, which then were used for the calculation of 3D surfaces which displayed on the visualization unit represent the 3D map of the object in three dimension and displayed graphically on the computer screen.
Some other methods and system also published and which were using pulsed laser light triggered by a function generating device and a gain-modulated camera. Objects are illuminated by the infrared pulsed laser. The gain of the camera that captures an image of the objects is linearly increased with respect to time. Thus, the light reflected from farther distances is amplified with a larger camera gain than that from nearer distances. The output image intensity of far objects is higher than that of near objects. The image intensity represents the distance to the object. With an increase in the speed of the camera gain modulation, the sensitivity of the depth measurement is increased.
Some methods are based on projecting a laser speckle pattern onto the object, and then analyzing the image of the pattern as projected on the object. For example, PCT International Publication WO 2007/043036, describes a system and method for object reconstruction, in which a coherent light source and a generator of a random speckle pattern projects onto the object a coherent random speckle pattern. An imaging unit detects the light response of the illuminated region and generates image data. Deformations of the pattern as projected on the object are compared with a reference image of the pattern in real-time and from the above comparison a 3D map of the object is calculated leading eventually to a 3D image of the object.
Variations of the above principles are used as in the PCT International Publication WO 93/03579 which describes a three-dimensional vision system in which one or two projectors establish structured light comprising two sets of parallel stripes having different periodicities and angles, while the U.S. Pat. No. 6,751,344 describes a method for optically scanning a subject in which the subject is illuminated with a matrix of discrete two-dimensional image objects, such as a grid of dots. Other methods involve projection of a grating pattern, as described, for example, in U.S. Pat. No. 4,802,759. Noticeable for all grid pattern based methods is that the precision of the reconstructed object suffers due to the discrete information extracted from the intersection points of the object and grid pattern.
Many computing applications such as electronic games, learning, multimedia, office applications or the like, or consumer electronic devices use various control methods either through computer commands or specific radiations for allowing users to manipulate game characters or other aspects of computer applications, or to control consumer electronic devices. Typically such controls are applied to the above computer applications or electronic devices using, for example, remotes controllers, keyboards, mice, joysticks or the like. Unfortunately, such control devices do not map directly to the spirit of the actual game or other application actions, or the spirit of naturally using electronic devices for which the controls are used. For example, a game control that causes a game character to either drive a car, fight in a fencing game, swing a baseball bat may not correspond to an actual motion of driving the car, fighting in a duel, or swinging the baseball bat. A new paradigm for human computer interface is therefore necessary.